Method and apparatus for producing an accessing composite data

ABSTRACT

An apparatus and method for producing composite data by a server computer involves deforming a template creating a mapping relationship between co-registered data and subject data, filtering the co-registered data, and mapping this filtered co-registered data according to the mapping data. A client computer requests the composite data from the server computer and transmits the subject data to the server computer. The client presents the received composite data to an operator and monitors the operators use of the composite data.

This application is a continuation of Ser. No. 09/382,594, filed Aug.25, 1999, issued as U.S. Pat. No. 6,253,210, which is a continuation ofpatent application Ser. No. 08/832,688, filed Apr. 11, 1997, issued asU.S. Pat. No. 5,970,499

BACKGROUND OF THE INVENTION

The present invention relates to information systems and methods, andmore particularly to data fusion systems.

Many applications require the contemporaneous assimilation of largeamounts of data Medical, military, and commercial applications requireoperators to engage in procedures and make decisions based on datadescribing various subjects represented by, for example, images,recorded sound, and text. Current technology does not present personnelwith a unified view of this subject data to allow then to use allavailable data to make informed decisions.

For example, a physician providing medical treatment, reviews image dataacquired in multiple modalities, such as magnetic resonance (“MR”),computed tomographic (“CT”), and X-ray images, medical journalsdescribing procedures, video images, such as ultrasound, and atlasesdescribing anatomical structures. A physician must therefore consultseveral sources to review the data necessary to provide patienttreatment. These sources may include multiple computer display terminalslocated in different parts of a hospital, hard copies of medical imagesprinted on film archived among thousands of images in a hospital filmlibrary or remote storage site, and volumes of journals located in thestacks of a hospital library. Also, the sources of data consulted bytreating physicians may include medical atlases containing thousands ofMR and CT scans of a cadaver corresponding to photographic images ofcross-sectional slices taken of various anatomical structures.

Usually data from these atlases and other sources are not correlatedwith each other. A cadaver image in an atlas does not usually have thesame geometry as a patient receiving treatment, so a physician mustmentally fuse the available data which requires correlating the dataretrieved from the various sources to develop a treatment plan or toprovide information during medical procedures. The difficulties offusing all of the available data increase if the physician mustassimilate the various data types while rendering treatment.

The World Wide Web (“WWW”) has recently made vast amounts of data storedon local and remote computers easily accessible through a graphicalcomputer interface. The WWW is a network of computers, connected by theInternet, sharing a common file structure and mark-up language forcreating files. The two most prevalent languages used to createmultimedia WWW files are the hypertext mark-up language (“HTML”) and thevirtual reality mark-up language (“VRML”). HTML is best suited forcreating files with text and two-dimensional image data, whereas VRML isdesigned for creating files containing images of three-dimensionalobjects. Both languages provide an easy way to combine image, text, andsound data in files accessible by “point-and-click,” computer mousedriven user interfaces called “browsers.”

A “browser” is a computer program that provides users access to filesstored on the WWW. The browser displays files on a computer screen andcan run programs, known as “applets,” indicating links to data in otherfiles on the WWW by, for example, underlining text or highlighting areasof an image. By selecting the underlined text or a highlighted image,the browser retrieves the linked data, allowing a user to view datastored on computers in the WWW without needing to know where theinformation is physically stored. Files can be joined using these“hyperlinks,” which give the name of the file along with an address fora computer storing the file. For example, the text or an image in a filestored on a computer in Switzerland can contain an embedded link to datastored on a computer in the United States. The WWW browser automaticallyrecognizes the linked file data type, so the linked file can be animage, an audio clip, a video, or even an executable computer program.For example, if the linked data is an audio clip, the browser will loada program that takes the audio clip and plays it through the speakers ofthe user's computer. A browser usually runs on a computer referred to asa “client,” while a computer known as a “server” hosts and produces WWWfiles requested by a client.

In particular, the WWW serves as a useful tool for navigating throughtwo- and three-dimensional image data. For example, an image can bedisplayed by the browser, and different parts of the image can be linkedto different files. But, for the most part, this WWW capability isprimarily used for providing simple menus of uncorrelated data availableon WWW computers. For example, a WWW computer will show an image ofpeople, cars, and boats. By clicking on the image of people, a user cango to on-line chat sessions with people, or by clicking on a boat image,a user gets information about boats.

The current technology is limited because there does not exist aninformation system that exploits the data navigation capabilities of theWWW to correlate data retrieved from diverse sources and then assimilatethe data into a useful form. For example, the tools available forinformation gathering in the WWW environment include database searchengines and expert systems that assist a user in describing theinformation sought. However, these tools only retrieve filescorresponding to a particular term or pertaining to certain designatedsubject matter. The retrieved files are not correlated with one another.

There is, therefore, a need for an information system that harnesses thepower of the technology associated with the WWW and other similarimage-based information retrieval systems to produce assimilatecomposite data in a form that operators can readily use.

SUMMARY OF THE INVENTION

The present invention is directed to a method and apparatus forproducing and accessing composite data containing co-registered andsubject data. The method for automatically producing composite dataincludes several steps, performed by a server computer. The stepsinclude: creating a mapping relationship between the co-registered dataand the subject data by mapping or deforming a template to fit thesubject data; filtering the co-registered data; and producing compositedata by mapping the filtered co-registered data to the subject dataaccording to the mapping relationship.

A method consistent with this invention is also directed to steps,performed in a client computer, including: requesting composite datafrom a server, computer; transmitting the subject data to the servercomputer; receiving the requested composite data from the servercomputer; presenting the received composite data to an operator; andmonitoring the operator's use of composite data.

An apparatus consistent with this invention for automatically producingcomposite data containing co-registered data and subject data includes:structure for creating a mapping relationship between the co-registereddata and the subject data by mapping or deforming a template to fit thesubject data; structure for filtering the co-registered data; andstructure for producing composite data by mapping the filteredco-registered data to the subject data according to the mappingrelationship.

Another apparatus consistent with the present invention automaticallypresents an operator with composite data containing co-registered dataand subject data. Such an apparatus includes: structure for requestingcomposite data from a server computer; structure for transmitting thesubject data to the server computer; structure for receiving therequested composite data from the server computer; structure forpresenting the received composite data to an operator; and structure formonitoring the operator's use of the received composite data.

Both the foregoing general description and the following detaileddescription are exemplary and explanatory and are intended to providefurther explanation of the invention as claimed.

DESCRIPTION OF THE FIGURES

The accompanying drawings provide a further understanding of theinvention. They illustrate embodiments of the invention and, togetherwith the description, explain the principled of the invention.

FIG. 1 is a block diagram of an apparatus for automatically producingcomposite data containing co-registered data and subject data consistentwith the present invention;

FIG. 2 is a block diagram of another apparatus for automaticallyproducing composite medical data containing co-registered medical dataand patient data consistent with the present invention;

FIG. 3 is flow diagram of a method for automatically producing compositedata containing co-registered data and subject data consistent with thepresent invention;

FIG. 4 is a schematic diagram of user interaction with the embodiment ofthe invention shown in the block diagram of FIG. 2;

FIG. 5 is an illustration of co-registered medical data used in anembodiment of the present invention consistent with the block diagram ofFIG. 2;

FIG. 6 is a display produced in accordance with the embodiment of thepresent invention shown in the block diagram of FIG. 2; and

FIG. 7 is a block diagram of a facility for providing composite data inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the preferred embodiments ofthis invention, examples of which are illustrated in the accompanyingdrawings.

To illustrate the principles of this invention, FIG. 1 shows subjectdata 100, template layer 102, deformation engine 104, mapping engine106, map 107, filtered, co-registered data 108, search and filter engine110, links 112, and co-registered databases 114. The term “layer”denotes a grouping of data types represented in, for example, a databaseon a server computer. To produce composite data 109, deformation engine104 deforms a template layer 102 to fit subject data 100 generating map107. When subject data 100 is multi-modal, multiple template layers canbe utilized to correlate co-registered data 114 to subject data 100 aswell as to correlate multi-modal subject data 100 to itself. A templatelayer contains reference data locations, or landmarks, used to correlatetemplate data with elements of subject data. Examples of landmarksinclude data representing points, lines, surfaces, volumes, or otherdefining features in image data.

Generally, deformation is the process of mapping one image to anotherimage where both images represent the same structure but have differentgeometric proportions and orientations. During deformation, mathematicaltransforms are applied to the images that perform the equivalent ofbending, stretching, and rotating template data to match subject data.For example, after deformation, template data in the form of a volumeimage of a generalized model of the human brain is manipulated so thatthe anatomical structure in this model has the same size, shape, andorientation as the subject data, the actual anatomy of a patientreceiving treatment. There are many techniques available forautomatically deforming one set of data to fit a target data set,including rule-based morphology, correlation of selected landmarks ineach data set, and a technique fusing selected landmarks and image data.One example of such technique appears in U.S. patent application Ser.No. 08/678,628 filed Jul. 10, 1996, now U.S. Pat. No. 6,009,212, whichis herein incorporated by reference.

Once template layer 102 is deformed to fit subject data 100, a mappingrelationship, map 107, is established whereby mapping engine 106 mapsco-registered data 114 to subject data 100 producing composite data 109.Co-registered data 114 represents a knowledge base providingsupplemental information about structures contained in subject data 100.The co-registered databases and template layer share a common coordinatesystem, so a data element representing a position in one co-registereddatabase is correlated with a data element representing that sameposition in each of the other co-registered databases. In an embodimentconsistent with the present invention, co-registered data 114 isco-registered using, for example, the deformation techniques describedabove. The mapping relationship obtained from deforming template layer102 to fit subject data 100 correlates the co-registered databasecoordinate system with a subject data coordinate system. Implementingthis mapping relationship, mapping engine 106 relates points in thesubject data coordinate system to corresponding points in theco-registered database coordinate system, providing a dynamic connectionbetween subject data 100 and co-registered databases 114.

Search and filter engine 110 controls which elements of co-registereddata 114 are mapped to subject data 100 and presented to an operator.Search and filter engine 110 can allow mapping engine 106 to map all ofa subset of co-registered data 114 to subject data 100. Links 112specify relationships among data elements across co-registered databases114 which are used by search and filter engine 110 to assimilateco-registered data according to a service request by an operatorincluding, for example, an indication of a region or regions of interestin subject data 100. The links 112 may be formed using an appropriatedatabase indexing strategy to assign key-word or concept search tags toassociated data elements.

FIG. 2 shows a preferred embodiment of the present invention forproducing composite patient data for medical treatment. Collectedpatient data 200, includes, for example, MR image 236, CT image 238, andlandmarks 240. Template layer 202 includes corresponding MR image 230,CT image 232, and landmark 234 templates. This embodiment also includesa deformation engine 204, mapping engine 206, search and filter engine210, links 212, and co-registered data 214.

Co-registered data 214 includes a medical atlas 228, text 226, computerprograms 225 (such as applets), labels and landmarks 224, images 222,audio and video clips 220, links to computers located on the WWW 218,and treatment plans 216. Co-registered data can also includeco-registered subject data. The forgoing list of co-registered datatypes is only provided as an example of the types of data that areuseful in practicing this invention in the context of providing medicaltreatment. Persons of ordinary skill will recognize that many other datatypes may also be useful. One of ordinary skill in the art will alsorecognize that two or more of the co-registered data types may reside ina single database.

FIG. 3 is a flow diagram of a method for producing composite medicaldata consistent with the invention. An operator of a system consistentwith this invention wishing to have composite data for medical treatmentfirst identifies and possibly collects patient data 200 using a clientcomputer (step 300). The collected patient data must have a modality(e.g., CT, MR, or X-ray) and protocol (e.g., slice thickness andresolution) that is compatible with a template in template layer 202.

The client computer then generates request 246 for composite data 244(step 304). Request 246 includes, for example, an operator identifier,security screening information, and treatment information. The clientcomputer also transmits or enables transmission of (from a radiologicaldatabase, for example) collected patient data 200 and associatedfiltering content 242 to the server computer (step 306).

Responding to client computer request 246 for composite data 244, theserver computer selects a template conforming with this request (step308). Example template, MR 230, employed by the present invention whenused for medical treatment includes a three-dimensional MR scan of thesame slice thickness, resolution, and collection protocol as the patientMR dataset 236. Associated with the selected template imagery is a setof landmarks 234 identifying anatomical structure in a region ofinterest for a particular surgical procedure. Next, deformation engine204 fits the selected template to patient data received from the clientcomputer. The process of deforming the selected template to fit thepatient data 200 creates a mapping relationship (map 248) relatingtemplate data space to a patient data space coordinate system (step310). Mapping engine 206 also uses map 248 to relate a point in thepatient data space coordinate system to an element of co-registered data214. Once the mapping relationship is determined by deforming theselected template, all co-registered data 214 can be mapped to patientdata 200. Note that if multi-modal patient data is used with multiplecorresponding templates, multiple maps 248 can be constructed that canthen be used to correlate the multi-modal patient data with each otherin addition to correlating co-registered data 214.

Search and filter engine 210 controls how much of co-registered data 214is included in the composite data. One reason for such control is thatcertain data types and/or data elements in co-registered data 214 maynot be relevant for some medical treatment. Search and filter engine 210responds to filtering context 242 and selects co-registered dataelements as appropriate for this filtering context (step 312) usinglinks 212 to identify related data elements. The filtering context canbe derived from data providing by the client computer during the initialrequest for composite data (step 304). A filtering context can also bederived from previously stored profiles or histories. The server thenproduces composite data using mapping engine 206 and map 248 to combinepatient data and filtered, co-registered data (step 314), producingcomposite data 244. The server then transmits composite data 244 and map248 to the client computer for presentation to an operator (step 318).The operator navigates the composite data by specifying a region ofinterest in the patient data using a browser interface (step 320). Theoperator may also use the browser interface to select highlighted textor other specified segments of the composite data, activating a link toa particular region of interest in the patient data.

Map 248 also allows an operator to access additional composite data(step 322). The server receives a request for additional co-registereddata and preferably an associated position in the patient datacoordinates from the client and subsequently retrieves and transmitsadditional co-registered data using mapping engine 206, map 248, searchand filter engine 210, and links 212 to additional co-registered data214. (repetition of steps 312-320).

An embodiment consistent with the present invention for use in themedical field links a radiologist's report to radiological imagery of apatient. Preferably, in an area of the radiologist's text reportstating, for example, “in the left parietal-occipital region is a 1.2 cmhypodense lesion with an irregular border that does not enhance oncontrast but is hyperintense of T2 . . . ”, selecting the highlightedword “lesion” activates a link to the patient imagery which highlightsthe particular sub-region discussed in the report text. Likewise, if theoperator selects the lesion site in the displayed patient imagery, thelink will be activated to display the section or sections of the textreport that discuss the selected region of interest.

Although the foregoing description of embodiments of the presentinvention specifically allocate certain operations to client and servercomputers, one of ordinary skill in that art will recognize that thedistribution of specific tasks between client and server computers canvary based on application requirements. Moreover, embodiments of thepresent invention with several client or server computers are within thescope of this invention. Furthermore, it is also consistent with thepresent invention that the client and server tasks can be performed in asingle computer.

In a preferred embodiment of the present invention, a graphical userinterface designed for browsing data presents composite data 244 to theoperator. The interface is preferably executed on networked computers.Computer program code that can be adapted to perform this browsingfunction includes Internet browsers designed to navigate the WWW, suchas Netscape's Navigator and Microsoft's Explorer, and equivalent programthat support links to data stored on networked computers.

FIG. 4 illustrates operator instruction associated with producingcomposite data in accordance with an embodiment of the presentinvention. FIG. 4 shows a database containing co-registered data 400 ofseveral data types, including video 402, text 404, waveforms 406,programs 407, still images 408, and segmentation labels 409; a mappingengine 410; a map 412; a set of links 414 among associated data elementsacross and within the co-registered databases; a patient data spacecoordinate system 416; and a universal (“atlas”) coordinate system 418common to all data stored in co-registered database 400. A physicianusing this embodiment of the present invention selects any point inpatient data space coordinate system 416 to retrieve co-registered data400 corresponding to the selected point.

In an example illustrating co-registered data in one embodiment of theinvention, still image database 408 contains MR images of a human head,database 406 contains recordings of waveforms produced by certainelectrical signals in the brain, video database 402 contains recordedmotion picture images of neurosurgical procedures or tutorials for theseprocedures, text database 404 contains short descriptive paragraphs orfull journal articles describing regions of the brain and relatedsurgical plans, database 407 contains programs for processing imagedata, and database 409 contains segmentation maps outlining brainstructures.

The patient data space coordinate system 416 is a frame of reference forpatient specific data. This coordinate system is provided by, forexample, an MRI of a patient's head or the surgical field surroundingthe patient during operation. Deformation engine 204 computes a mappingrelationship relating template layer data points in atlas coordinatesystem 418 to patient data points in patient data space coordinatesystem 416. Mapping engine 410 uses this computed mapping relationshipto transform co-registered data 400 mapped to atlas coordinate system418 to patient data space coordinate system 416.

After mapping, a physician has available during a surgical procedurecomposite data adapted to the patient's anatomy. This composite data isa representation of (1) a patient's anatomy comprising patient specificdata acquired before or during a medical procedure, and (2) data fromone or more of the co-registered databases 400.

Map 412 provides a virtual grid overlaying the patient data spacecoordinate system 416 allowing an operator to position a pointing devicein the patient data to retrieve co-registered data. Selecting a positionin map 412 retrieves co-registered data correlated with the selectedposition by mapping engine 410 through links 414.

In one embodiment of the invention, map 412 contains a number ofpositions corresponding to the number of positions in patient data spacecoordinate system 416 detectable by a surgical navigation system (see,e.g., U.S. Pat. No. 5,383,454). A map position is selected according tothe location of a surgical probe in patient data space coordinate system416 during a medical procedure. For example, during neurosurgery asurgeon placing the probe at a patient's ventricle activates a mapposition corresponding to the probe position in the ventricle. Theactivated map position is communicated to mapping engine 410 whichqueries co-registered databases 400 for data corresponding to the mapposition in the ventricle. This corresponding co-registered data isdeformed to fit the patient's anatomy and combined with patient specificdata giving the surgeon composite data related to the patient'sventricle, containing more information than the patient data alone.

A further illustration of the types of data a physician may requireduring neurosurgery or during surgical planning is shown in FIG. 5. Thisfigure contains illustrations of six data types available to a surgeonin an embodiment of the present invention including a cross-sectionalimage 502, a medical journal article 504, electroencephalographwaveforms 506, computer programs 507, video images of the brain 508, anda segmentation map identifying the different regions of the brain 510.Because these data sources have been co-registered to a common atlascoordinate system, a point, such as point 500 in the cerebellum in brainimage 502 from database 408, has a corresponding point in each of theother data types in co-registered database 400. For example, a point 500in text database 404 corresponds to article 504 on new surgicaltechniques involving the cerebellum. Point 500 in waveform database 400corresponds to recorded waveforms 506 produced by the brain at thislocation. Point 500 in program database 407 corresponds to appletprogram 507, which is used to enhance visualization of brain image 502.Point 500 in video database 402 corresponds to video clips 508 of thebrain at this location. Point 500 in segmentation map database 409corresponds to a point within segmentation map 510.

Each of these examples of data need not be acquired from the patientcurrently receiving treatment. For example, the data may come fromdigital anatomical atlases, libraries of cadaver images, or researchdatabases produced by projects such as the “Visible Human” researchsponsored by the National Library of Medicine. Data available in aco-registered database would include, for example:

1. Anatomic

Magnetic Resonance Imaging

Computer Technology

Magnetic Resonance Angiography

Ultra-Sound

Slice photographic images

Sulci/Gyri traces

2. Functional

Positron Emission Tomography

Single Photon Emission Computer Tomography

Functional Magnetic Resonance images

Electroencephalograph

Magnetoencephalography

3. Symbolic

Structure name

Structure size

Structure function

Structure related text cues

Structure related video cues

Structure related audio cues

Structure related labels

Histology

Morphological data

4. Multimedia

Video Footage of procedures

Training Videos

Conference, Journal Articles

Statistics

5. Computer Programs

Applets

Data Analysis

Automated Structural Segmentation

Image Enhancement and Visualization

These data need not be located in a single database. One of ordinaryskill in the art will recognize that individual co-registered databasesmay be distributed among several databases accessible through, forexample, local area computer networks or wide area computer networksconnecting local and distributed computers. The combination of thedifferent data components of the co-registered databases produces ageneric data model of human anatomy where each data element iscorrelated through a common coordinate system with corresponding dataelements of other types. When planning or providing medical treatment,the invention produces composite data linking co-registered data of ageneric model to the specific anatomy of a patient receiving treatment.Examples of other embodiments consistent with the present invention forproducing composite data for medical diagnosis, planning, and treatmentinclude, but are not limited to, the following.

1. Diagnostic Radiology—co-registered patient magnetic resonance, X-ray,and/or computed tomography imagery are linked to text data such asradiologists's reports, patient case history files, and relevantconference/journal articles. Sequential scans of a patient areco-registered for tracking the growth or reduction of lesions. Dataanalysis programs are linked to the composite data for computation ofquantitative data measurements for planning and monitoring of treatmentprogress. Co-registered multi-modal patient image data and relevantco-registered data are presented in a common, easy-to-use presentationscheme.

2. Radiation Treatment Planning—Three-dimensional segmented atlases aremapped to patient data to produce an object-based model of lesions,targets, and major organs and other critical structures. The patientdata with associated object information is utilized by a treatmentplanning program for computing optimized radiation delivery strategiesfrom target and critical structure information.

3. Neurosurgical Targeting—cranial patient imagery is mapped toneurosurgical atlas information containing coordinates and shapes ofsurgical targets and surrounding neuroanatomic structures. Structuralinformation is linked to audio files for use in-surgery withmicrorecording probes. Links to statistical databases provideinformation relating regions of interest to procedures and successrates.

FIG. 6 shows how one embodiment of the present invention presentscomposite data to an operator on a computer display having severalwindows. Window 604 contains patient data in the form of athree-dimensional MR brain scan. Windows 602, 606 and 608 contain axialsagittal, and coronal photographic section data, respectively, from theVisible Human data set, which are co-registered to the patient data bydeformation engine 204. Window 610 presents co-registered atlas data toan operator. By positioning cross-hairs 612 in Window 604 at a desiredpoint 600, the corresponding point 600 in each of the images in windows602, 606, and 608 is identified automatically by the location ofcross-hairs in those windows. Here, for example, the operator selectedpoint 600 in window 604 corresponding to a region of the brain known asthe putamen. Atlas data containing views of the putamen are displayed inwindow 610 with highlighted pointers indicating the putamen in each ofthe atlas images. The operator can also choose to play a movie showingvideo images of the putamen by pushing “play movie” button 614.Alternatively the operator may select the word “putamen” in window 610and cross-hairs 612 will indicate the position of the putamen in patientdata window 604 and the Visible Human “atlas” data windows 602, 606,608.

FIG. 7 is a block diagram of a facility consistent with the presentinvention for providing composite data across a computer network toconsumers under a service contract. In FIG. 7, solid lines indicate apath for both control and data flow and dotted lines indicate data flowonly. Facility 700 is preferably connected to a wide area network, suchas Internet 701, through firewall 702. Firewall 702 is a computer thatmonitors all data traffic into and out of facility 700 to preventunauthorized access to the facility. World Wide Web page 704 provides agraphical user interface to access facility 700. Facility 700 alsoincludes customer account manager 710, which controls functionsavailable to customers with service contracts authorizing access tofacility 700.

User login authentication is performed by customer account manager 710and control is passed to one of three processes, service request manager706, customer database manager 708, or results manager 712 depending onthe service that the customer chooses. Customers that wish to initiate anew request for composite data are passed to service request manager706. After successful completion of a composite data request, thecustomer's account is billed and the status of any pending requests isprovided. Customers that wish to view the composite data generated inresponse to the request are passed to results manager 712. Informationpertaining to a customer's account (e.g., billing information, changingpasswords, user preferences, etc.) may be obtained by submitting queriesto customer database manager 708.

Service request manager 706 initiates service requests and controls thecomputational operations required for anatomic mapping. The progress ofa service request is tracked and reported to customer account manager710. Customer database manager 708 administers a database that containscustomer account data (not shown). Customer database manager 708 isresponsible for controlling and backing up the customer database and italso processes queries from customer account manager 710, servicerequest manager 706, and results manager 712. Results manager 712integrates the results generated by service request manager 706 withcontext-specific medical knowledge. Results manager 712 receivesinformation from search and filter engine 720 and mapping engine 722specific to application requirements. Some results may be provided asvisual representations (e.g., mapped segmented structures in a brainimage) while others may be expressed in numeric form (e.g., coordinatesof a neurosurgical target).

Processor 714 checks patient data associated with a server request tomake sure that the patient data is in the correct format and that theservice request is appropriate. Support personnel 724 confirm the checkperformed by preprocessor 714. Any inconsistencies or anomalies that arefound are reported to service request manager 706. Similarly, postprocessor 718 checks the results of a service request. Support personnel724 confirm the check performed by postprocessor 718. Anyinconsistencies or anomalies that are found are reported to servicerequest manager 706.

Facility 700 also includes deformation engine 716, search and filterengine 720, and mapping engine 722. Deformation engine 716 computesatlas-to-patient transformations requested by service request manager706. Search and filter engine 720 processes customer, patient, andprocedure contextual data and integrates relevant atlas informationaccording to application requirements. Both results manager 712 andservice request manager 706 initiate search and filter engine 720operations. Following a request from results manager 712, mapping engine722 applies the results of the deformation process mapping an atlas tothe patient data.

Patient database manager 726 administers a database that containspatient data and corresponding transformation computed by deformationengine 716. Patient database manager 726 serves queries from deformationengine 716, search and filter engine 720, and mapping engine 722 and isalso responsible for controlling and backing up the patient database(not shown).

Atlas database manager 728 administers a database that contains theatlas data (not shown). Atlas database manager 723 serves queries fromdeformation engine 716, search and filter engine 720, and mapping engine722 and is also responsible for controlling and backing up the atlasdatabase. Atlas database manager 728 can also perform and/or manage theindexing of the co-registered data databases.

While there has been illustrated and described what are at presentconsidered to be preferred embodiments and methods of the presentinvention, persons skilled in the art will understand that variouschanges and modifications may be made, and equivalents may besubstituted without departing from the scope of the invention.

In addition, many modifications may be made to adapt a particularelement, technique or implementation to the teachings of the presentinvention without departing from the central scope of the invention. Forexample, disclosed elements may be implemented in hardware, computerprogram code, or a combination of both hardware and computer programcode. Moreover, elements depicted and described separately may becombined and implemented in a single element. Therefore, this inventionis not limited to the particular embodiments and methods disclosed, butincludes all embodiments falling within the scope of the appendedclaims.

We claim:
 1. A method for presenting an operator with composite medicaldata containing subject data and co-registered data representing aknowledge base providing supplemental information about structurecontained in the subject data, said method comprising: creating amapping relationship between the subject data and the co-registered databy deforming a template corresponding to the subject data; linking,based on the mapping relationship, co-registered data in co-registereddatabases that include dose curve data and medical atlas data; filteringthe co-registered data to allow mapping of at least a portion of theco-registered data; mapping the filtered co-registered data to thesubject data according to the mapping relationship to produce thecomposite data; and presenting the operator the composite medical dataincluding the dose curve data and medical atlas data.
 2. The method ofclaim 1, wherein the step of linking the medical atlas data to the dosecurve data includes co-registering the dose curve data and the medicalatlas data to a common atlas coordinate system.
 3. The method of claim1, wherein the dose curve data is a graphical representation of computeddosages.
 4. The method of claim 1, wherein the dose curve data is agraphical representation of calculated therapy settings.
 5. The methodof claim 1, wherein the step of mapping includes the step of relating apoint in a subject data space coordinate system to an element ofco-registered data.
 6. The method of claim 1, wherein the presentingstep includes displaying the composite medical data as a VRML display.7. The method of claim 1, wherein the step of presenting the compositemedical data includes composing the composite medical data on a servercomputer and transmitting the composite medical data to a clientcomputer for presentation to the operator.
 8. An apparatus forpresenting an operator with composite medical data containing subjectdata and co-registered data representing a knowledge base providingsupplemental information about structure contained in the subject data,said apparatus comprising: a deformation engine that creates a mappingrelationship between co-registered data, in co-registered databases, andsubject data by deforming a template corresponding to the subject data;links that, based on the mapping relationship, specify a relationship inco-registered databases that include medical atlas data and dose curvedata; a filtering engine that filters the co-registered data to allowmapping of at least a portion of the co-registered data; a mappingengine that maps the filtered co-registered data according to themapping relationship to produce the composite medical data; and agraphical display that presents to the operator the composite medicaldata, including the dose curve data and the medical atlas data.
 9. Theapparatus of claim 8, wherein the links are created by co-registeringthe dose curve data and the medical atlas to a common atlas coordinatesystem.
 10. The apparatus of claim 8, wherein the dose curve data is agraphical representation of computed dosages.
 11. The apparatus of claim8, wherein the dose curve data is a graphical representation ofcalculated therapy settings.
 12. The apparatus of claim 8, wherein thegraphical display is located at a client computer and the compositemedical data is composed on a server computer and transmitted to theclient computer for presentation to the operator.
 13. The apparatus ofclaim 8, wherein the graphical display presents the dose curve data andthe medical atlas data as a VRML image.
 14. An apparatus for presentingan operator with composite medical data containing subject data andco-registered data representing a knowledge base providing supplementalinformation about structure contained in the subject data, and apparatuscomprising: a deformation engine that creates a mapping relationshipbetween the subject data and the co-registered data, in co-registereddatabases, by deforming a template corresponding to the subject data; afiltering engine that filters the co-registered data corresponding to amap position supplied by a surgical navigation system to allow at leasta portion of the corresponding co-registered data to be mapped; amapping engine that maps the filtered co-registered data correspondingto the map position to the subject data according to the mappingrelationship to produce the composite medical data; and a display thatpresents the operator with the composite medical data composed of thecorresponding co-registered data mapped to the subject data.
 15. Theapparatus of claim 14, wherein the surgical navigation system selectsthe map position based on a location of a surgical probe in a subjectspace.
 16. The apparatus of claim 14, wherein the display presents thecomposite medical data as an image.
 17. The apparatus of claim 14,wherein the display presents the composite medical data as text.
 18. Theapparatus of claim 14, wherein the display presents the compositemedical data as a VRML image.
 19. An method for presenting an operatorwith composite medical data containing subject data and co-registereddata representing a knowledge base providing supplemental informationabout structure contained in the subject data, said method comprising:creating a mapping relationship between the subject data and theco-registered data, in co-registered databases, by deforming a templatecorresponding to the subject data; filtering the co-registered datacorresponding to a map position supplied by a surgical navigation systemto allow at least a portion of the corresponding co-registered data tobe mapped; mapping the filtered co-registered data corresponding to themap position to the subject data according to the mapping relationshipto produce the composite medical data; and displaying to an operator arepresentation of the composite medical data composed of theco-registered corresponding data mapped to the subject data.
 20. Themethod of claim 19, wherein supplying the map position includes thesurgical navigation system selecting a map position based on a locationof a surgical probe in a subject space.
 21. The method of claim 19,wherein the displaying step includes presenting the composite medicaldata as an image.
 22. The method of claim 19, wherein the displayingstep includes presenting the composite medical data as text.
 23. Themethod of claim 19, wherein the displaying step includes presenting thecomposite medical data as a VRML image.
 24. An apparatus for presentingan operator with composite medical data containing subject data andco-registered data representing a knowledge base providing supplementalinformation about structure contained in the subject data, saidapparatus comprising: means for creating a mapping relationship betweenthe subject data and the co-registered data, in co-registered databases,by deforming a template corresponding to the subject data; means forfiltering the co-registered data corresponding to a map positionsupplied by a surgical navigation system to allow at least a portion ofthe corresponding co-registered data to be mapped; means for mapping thefiltered co-registered data corresponding to the map position to thesubject data according to the mapping relationship to produce thecomposite medical data; and means for presenting the operator withcomposite medical data composed of the corresponding co-registered datamapped to the subject data.
 25. The apparatus of claim 24, wherein themeans for filtering comprises means for use with a surgical navigationsystem for selecting the map position based on a location of a surgicalprobe.
 26. The apparatus of claim 24, wherein the means for presentingthe composite medical data comprises means for displaying the compositemedical data image.
 27. The apparatus of claim 24, wherein the means forpresenting composite medical data comprises means for displaying thecomposite medical data as a VRML image.
 28. The method of claim 24,wherein the means for presenting the composite medical data comprisesmeans for displaying the composite medical data as text.
 29. Anapparatus for tracking change of brain structures and lesions withcomposite medical data, comprising: a deformation engine that creates amapping relationship between scans of a subject's brain structures andlesions and subject data by deforming a template corresponding to thesubject data; a filtering engine that filters the scans to allow themapping of at least a portion of the scans to the subject data; amapping engine that maps the scans that have been filtered to thesubject data to form the composite medical data; and a graphical displaythat presents the composite medical data composed of the scans of thesubject's brain structures and lesions, and the subject data.
 30. Theapparatus of claim 29, wherein the scans are sequential scans of asubject's brain structure and lesions.
 31. The apparatus of claim 29,wherein the mapping engine produces the composite medical data byrelating points in a subject space coordinate system to scans in aco-registered database coordinate system.
 32. The apparatus of claim 29,wherein the graphical display presents the composite medical data to theoperator as a VRML image.
 33. An apparatus for tracking change of brainstructures and lesions with composite medical data, comprising: meansfor creating a relationship between scans of a subject's brainstructures and lesions and subject data by mapping or deforming atemplate corresponding to the subject data; means for filtering thescans to allow at least a portion of the scans to be mapped to thesubject data; means for mapping the scans that have been filtered to thesubject data to form the composite medical data; and means fordisplaying the composite medical data composed of the scans of thesubject's brain structures and lesions, and the subject data.
 34. Theapparatus of claim 33, wherein the means for creating the mappingrelationship comprises means for creating the mapping relationshipbetween sequential scans of the subject's brain structures and lesionsand the subject data.
 35. The apparatus of claim 33, wherein the meansfor mapping the scans comprises means for producing the compositemedical data by relating points in a subject space coordinate system tothe scans in a co-registered database coordinate system.
 36. Theapparatus of claim 35, wherein the means for displaying includes meansfor presenting the composite medical data to the operator as a VRMLimage.
 37. A method for change of brain structures and lesions withcomposite medical data, comprising: creating a relationship betweenscans of a subject's brain structures and lesions and the subject databy mapping or deforming a template corresponding to the subject data;filtering the scans to allow the mapping of at least a portion of thescans to the subject data; mapping the scans that have been filtered tothe subject data to form the composite medical data; and displaying thecomposite medical data composed of the scans of the subject's brainstructures and lesions, and the subject data.
 38. The method of claim37, wherein the relationship is created between sequential scans of thesubject's brain structures and lesions and the subject data.
 39. Themethod of claim 37, wherein the mapping step includes forming thecomposite medical data by relating points in a subject space coordinatesystem to the scans in a co-registered database coordinate system. 40.The method of claim 37, wherein the displaying step includesrepresenting the composite medical data to the operator as a VRML image.41. An apparatus for linking a region of interest in a subject data setto a statistical database that relates the region of interest toinformation on procedures, comprising: a deformation engine that createsa mapping relationship between the statistical database and the subjectdata set by deforming a template corresponding to the subject data set;a filtering engine that filters the statistical database to allow atleast a portion of the statistical database to be mapped to the subjectdata; a mapping engine that maps the filtered statistical database,which contains the information on procedures, to the subject data toform the composite medical data; and a display that presents thecomposite medical data composed of the subject data and information onprocedures.
 42. The apparatus of claim 41, wherein the procedureincludes the use of electrodose settings.
 43. The apparatus of claim 41,wherein the display presents the composite medical data composed of thesubject data and information on procedures as text and a VRML image. 44.The apparatus of claim 41, wherein the mapping engine forms thecomposite medical data by relating points in a subject space coordinatesystem to the statistical database in a co-registered databasecoordinate system.
 45. The apparatus of claim 41, wherein the displaypresents the composite medical data as graphics and text.
 46. Anapparatus for linking a region of interest in subject data to astatistical database that relates the region of interest to informationon procedures, comprising: means for creating a mapping relationshipbetween the statistical database and the subject data; means forfiltering the statistical database to allow at least a portion of thestatistical database to be mapped to the subject data; means for mappingthe filtered statistical database, which contains the information onprocedures, to the subject data to form the composite medical data; andmeans for presenting the composite medical data composed of the subjectdata and information on procedures.
 47. The apparatus of claim 46,wherein the means for presenting includes means for presenting aprocedure that uses electrodose settings.
 48. The apparatus of claim 46,wherein the means for mapping includes means for forming the compositemedical data by relating points in a subject space coordinate system tothe statistical database in a co-registered database coordinate system.49. The apparatus of claim 46, wherein the means for presenting includesmeans for displaying the composite medical data as graphics and text.50. The apparatus of claim 46, wherein the means for presenting includesmeans for displaying the composite medical data as a VRML image andtext.
 51. A method for linking a region of interest in subject data to astatistical database that relates the region of interest to informationon procedures, comprising: creating a mapping relationship between thestatistical database and a subject data set by deforming a templatecorresponding to the subject data; filtering the statistical database tomap at least a portion of the statistical database to the subject dataset; mapping the filtered statistical database, which contains theinformation on procedures, to the subject data to form the compositemedical data; and presenting the composite medical data composed of thesubject data and information on procedures that correspond with aselected point.
 52. The method of claim 51, wherein the step ofpresenting includes displaying a procedure that uses electrodosesettings.
 53. The method of claim 51, wherein the step of mappingincludes forming the composite medical data by relating points in asubject space coordinate system to the statistical database in aco-registered database coordinate system.
 54. The apparatus of claim 51,wherein the step of presenting includes displaying the composite medicaldata as graphics and text.
 55. The apparatus of claim 51, wherein thestep of presenting includes displaying the composite medical data as aVRML image and text.
 56. An apparatus for creating an object based modelof an area of interest within a subject, comprising: a deformationengine that creates a mapping relationship between segmented atlases ofa plurality of dimensions and a subject data set by deforming a templatecorresponding to the subject data set; a filtering engine that filtersthe segmented atlases to allow at least a portion of the segmentedatlases to be mapped to the subject data set; a mapping engine that mapsthe segmented atlases that have been filtered to the subject data set toform the composite medical data, which includes the object based model;and a display that presents the composite medical data composed of theobject based model of the area of interest.
 57. The apparatus of claim56, wherein the object based model of the area of interest is arepresentation of one of lesions, targets, and brain structures.
 58. Theapparatus of claim 56, wherein the display presents the compositemedical data composed of the object based model of the area of interestdisplayed as a graphical image.
 59. The apparatus of claim 56, whereinthe display represents the composite medical data composed of the objectbased model of the area of interest displayed as a VRML image.
 60. Amethod for creating an object based model of an area of interest withina subject, comprising: creating a relationship between segmented atlasesof a plurality of dimensions and a subject data set by deforming atemplate corresponding to the subject data set; filtering the segmentedatlases to allow at least a portion of the segmented atlases to bemapped to the subject data set; mapping the segmented atlases that havebeen filtered to the subject data set to form the composite medicaldata, which includes the object based mode; and displaying the compositemedical data composed of the object based model of the area of interest.61. The method of claim 60, wherein the displaying step includes agraphic representation of the object based model of one of lesions,targets, and brain structures.
 62. The method of claim 60, wherein thedisplaying step includes a VRML image of the object based model of oneof lesions, targets, and brain structures.
 63. An apparatus for creatingan object based model of an area of interest within a patient saidapparatus for use with one or more co-registered databases representinga knowledge base providing supplemental information about structurecontained in the subject data, said apparatus comprising: means forcreating a relationship between segmented atlases of a plurality ofdimensions and a subject data set by deforming a template correspondingto the subject data set; means for filtering the segmented atlases toallow at least a portion of the segmented atlases to be mapped to thesubject data set; means for mapping the segmented atlases that have beenfiltered to the subject data set to form the composite medical data,which include the object based model; and means for displaying thecomposite medical data composed of the object based model of the area ofinterest.
 64. The apparatus of claim 63, wherein the means fordisplaying includes means for providing a graphic representation of theobject based model of one of lesions, targets, and brain structures. 65.The apparatus of claim 63, wherein means for displaying include meansfor providing a VRML image of the object based model of one of lesions,targets, and brain structures.
 66. An apparatus that uses compositemedical data to supply a navigation system with a surgical plan, saidapparatus for use with one or more co-registered databases representinga knowledge base providing supplemental information about structurecontained in the subject data, said apparatus comprising: a deformationengine that creates a relationship between segmented atlases of aplurality of dimensions and a subject data set by deforming a templatecorresponding to the subject data set; a mapping engine that receives amap position from the navigation system and queries at least one of theco-registered databases for data corresponding to the map position andsupplies the data corresponding to the map position to the navigationsystem.
 67. The apparatus of claim 66, wherein the mapping enginesupplies a surgical plan to the navigation system based on atlasinformation.
 68. An apparatus that uses composite medical data to supplya navigation system with a surgical plan, said apparatus for use withone or more co-registered databases representing a knowledge baseproviding supplemental information about structure contained in thesubject data, said apparatus comprising: means for creating arelationship between segmented atlases of a plurality of dimensions anda subject data set by deforming a template corresponding to the subjectdata set; means for receiving a map position from the surgicalnavigation system and querying at least one of the co-registereddatabases for data corresponding to the map position; and means forsupplying the data corresponding to the map position to the navigationsystem.
 69. The apparatus of claim 68, wherein the means for supplyinginclude means for providing a surgical plan to the navigation systembased on atlas information.
 70. An apparatus that in conjunction with anavigation system presents an operator with composite medical datacontaining subject data and co-registered data representing a knowledgebase providing supplemental information about structure contained in thesubject data, said apparatus comprising: a deformation engine thatcreates a relationship between the subject data and the co-registereddata, in at least one co-registered database, by deforming a templatecorresponding to the subject data; a mapping engine that receives a mapposition from the navigation system and queries the at least oneco-registered database for data corresponding to the map position; afiltering engine that filters the data corresponding to the map positionto allow at least a portion of the corresponding data to be mapped tothe subject data; and a display on the navigation system that presentsthe operator with the composite medical data composed of thecorresponding data mapped to the subject data.
 71. The apparatus ofclaim 70, wherein the display is a VRML display.
 72. The apparatus ofclaim 70, wherein the display is a graphical display.
 73. The apparatusof claim 70, wherein the display on the navigation system only displaysthe boundaries of an atlas found in the at least one co-registereddatabase.
 74. A method for presenting composite medical data containingsubject data and co-registered data representing a knowledge baseproviding supplemental information about structure contained in thesubject data on a navigation system, said method comprising: generatinga relationship between the subject data and the co-registered data, inat least one co-registered database, by deforming a templatecorresponding to the subject data; receiving a map position from thenavigation system and querying the at least one co-registered databasefor data corresponding to the map position; filtering the datacorresponding to the map position to allow at least a portion of thecorresponding data to be mapped to the subject data; and presenting onthe navigation system composite medical data composed of thecorresponding data mapped to the subject data.
 75. The method of claim74, wherein the step of presenting includes displaying the compositemedical data as a VRML display.
 76. The method of claim 74, wherein thestep of presenting includes displaying the composite medical data as agraphical display.
 77. The method of claim 74, wherein the step ofpresenting includes only displaying on the navigation system theboundaries of an atlas found in the at least one co-registered database.78. An apparatus that in conjunction with a navigation system presentsan operator with composite medical data containing subject data andco-registered data representing a knowledge base providing supplementalinformation about structure contained in the subject data, saidapparatus comprising: means for creating a relationship between thesubject data and the co-registered data, in at least one co-registereddatabase, by deforming a template corresponding to the subject data;means for receiving a map position from the navigation system andquerying the at least one co-registered database for data correspondingto the map position; means for filtering the data corresponding to themap position to allow at least a portion of the corresponding data to bemapped to the subject data; and means for presenting on the navigationsystem composite medical data composed of the corresponding data mappedto the subject data.
 79. The apparatus of claim 78, wherein the meansfor presenting includes means for displaying the composite medical dataas a VRML display.
 80. The apparatus of claim 78, wherein the means forpresenting includes means for displaying the composite medical data as agraphical display.
 81. The apparatus of claim 78, wherein the means forpresenting includes means for only displaying on the navigation systemthe boundaries of an atlas found in the at least one co-registereddatabase.
 82. An apparatus for automatically presenting an operator withcomposite medical data containing patient data and co-registered datarepresenting a knowledge base providing supplemental information aboutstructure contained in the patient data, said apparatus comprising:means for requesting composite medical data from a server computer;means for transmitting the patient data to the server computer; meansfor transmitting the requested composite medical data from the servercomputer; and means including a surgical navigation system forpresenting the received composite medical data to an operator.
 83. Anapparatus for automatically presenting an operator with compositemedical data containing patient data and co-registered data representinga knowledge base providing supplemental information about structurecontained in the patient data, said apparatus comprising: meansincluding a surgical navigation system for requesting composite medicaldata from a server computer; means for transmitting that patient data tothe server computer; means for transmitting the requested compositemedical data from the server computer; and means for presenting thereceived composite medical data to an operator.
 84. A method forautomatically producing composite data containing subject data andco-registered data representing a knowledge base providing supplementalinformation about structure contained in the subject data, said methodcomprising the steps, performed in a computer having a mapping engine, asearch and filter engine, and a deformation engine, of: creating amapping relationship between the co-registered data and the subject databy deforming a template to fit the subject data; filtering theco-registered data; and mapping the filtered co-registered data to thesubject data according to the mapping relationship to produce thecomposite data.
 85. The method of claim 84, wherein the co-registereddata contains co-registered subject data.
 86. The method of claim 84,wherein the step of mapping includes the substep of relating a point ina subject data space coordinate system to an element of co-registereddata.
 87. The method of claim 84, wherein the step of mapping includesthe substep of relating an element of co-registered data to a point in asubject data space coordinate system.
 88. The method of claim 84,further comprising the step of receiving a request for the compositedata from an operator.
 89. The method of claim 88, further comprisingthe step of processing said request using a filtering context for thecomposite data.
 90. The method of claim 84, further comprising the stepof communicating the composite data to a display on the computer. 91.The method of claim 84, further comprising the step of generating a maprepresenting the relationship between the subject data and theco-registered data with the computer.
 92. The method of claim 84,further comprising the step of filtering the co-registered dataaccording to a filtering context.
 93. A computer having a mappingengine, a deformation engine, and a search and filter engine forautomatically producing a composite medical data containing subject dataand co-registered medical data representing a knowledge base providingsupplemental information about structure contained in the subject data,said computer comprising: means for creating a mapping relationshipbetween the co-registered medical data and the subject data by mappingor deforming a template to fit the subject data; means for filtering theco-registered medical data; means for mapping the filtered co-registeredmedical data to the subject data according to the mapping relationshipto produce the composite medical data.
 94. The computer of claim 93,wherein the co-registered medical data contains co-registered subjectdata.
 95. The computer of claim 93, wherein the means for creating amapping relationship comprises means for relating a point in a subjectdata space coordinate system to an element of co-registered medicaldata.
 96. The computer of claim 93, wherein the means for mappingfurther comprises means for relating an element of co-registered medicaldata to a point in a subject data space coordinate system.
 97. Thecomputer of claim 93, further comprising means for receiving a requestfor the composite medical data from an input device.
 98. The computer ofclaim 93, further comprising means for processing a filtering context;and means for processing a request, including a filtering context, forthe composite medical data.
 99. The computer of claim 93, furthercomprising means for displaying the composite medical data to anoperator.
 100. The computer of claim 93, further comprising means forcommunicating a map representing the relationship between the subjectdata and the co-registered medical data to an operator using thecomputer.
 101. The computer of claim 93, wherein the means for producingcomposite medical data comprises means for filtering the co-registeredmedical data according to a filtering context.